
Neural Networks 

David Kriesel 

dkriesel.com 



Download location: 

http://www.dkriesel.com/en/science/neural_networks 




NEW - for the programmers: 

Scalable and efficient NN framework, written in JAVA 
http://www.dkriesel.com/en/tech/snipe 



dkriesel.com 



In remembrance of 

Dr. Peter Kemp, Notary (ret.), Bonn, Germany. 



in 



D. Kriesel - A Brief Introduction to Neural Networks (ZETA2-EN) 



A small preface 



"Originally, this work has been prepared in the framework of a seminar of the 
University of Bonn in Germany, but it has been and will be extended (after 
being presented and published online under www.dkriesel.com on 
5/27/2005). First and foremost, to provide a comprehensive overview of the 
subject of neural networks and, second, just to acquire more and more 
knowledge about I^T^X . And who knows - maybe one day this summary will 

become a real preface!" 

Abstract of this work, end of 2005 



The above abstract has not yet become a 
preface but at least a little preface, ever 
since the extended text (then 40 pages 
long) has turned out to be a download 
hit. 

Ambition and intention of this 
manuscript 

The entire text is written and laid out 
more effectively and with more illustra- 
tions than before. I did all the illustra- 
tions myself, most of them directly in 
IAT^X by using XYpic. They reflect what 
I would have liked to see when becoming 
acquainted with the subject: Text and il- 
lustrations should be memorable and easy 
to understand to offer as many people as 
possible access to the field of neural net- 
works. 

Nevertheless, the mathematically and for- 
mally skilled readers will be able to under- 



stand the definitions without reading the 
running text, while the opposite holds for 
readers only interested in the subject mat- 
ter; everything is explained in both collo- 
quial and formal language. Please let me 
know if you find out that I have violated 
this principle. 

The sections of this text are mostly 
independent from each other 

The document itself is divided into differ- 
ent parts, which are again divided into 
chapters. Although the chapters contain 
cross-references, they are also individually 
accessible to readers with little previous 
knowledge. There are larger and smaller 
chapters: While the larger chapters should 
provide profound insight into a paradigm 
of neural networks (e.g. the classic neural 
network structure: the perceptron and its 
learning procedures), the smaller chapters 
give a short overview - but this is also ex- 
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plained in the introduction of each chapter. 
In addition to all the definitions and expla- 
nations I have included some excursuses 
to provide interesting information not di- 
rectly related to the subject. 

Unfortunately, I was not able to find free 
German sources that are multi-faceted 
in respect of content (concerning the 
paradigms of neural networks) and, nev- 
ertheless, written in coherent style. The 
aim of this work is (even if it could not 
be fulfilled at first go) to close this gap bit 
by bit and to provide easy access to the 
subject. 

Want to learn not only by 
reading, but also by coding? 
Use SNIPE! 

SNIPE 1 is a well-documented JAVA li- 
brary that implements a framework for 
neural networks in a speedy, feature-rich 
and usable way. It is available at no 
cost for non-commercial purposes. It was 
originally designed for high performance 
simulations with lots and lots of neural 
networks (even large ones) being trained 
simultaneously. Recently, I decided to 
give it away as a professional reference im- 
plementation that covers network aspects 
handled within this work, while at the 
same time being faster and more efficient 
than lots of other implementations due to 

1 Scalable and Generalized Neural Information Pro- 
cessing Engine, downloadable at http://www. 
dkriesel.com/tech/snipe online JavaDoc at 
http : / / snipe . dkriesel . com 



the original high-performance simulation 
design goal. Those of you who are up for 
learning by doing and/or have to use a 
fast and stable neural networks implemen- 
tation for some reasons, should definetely 
have a look at Snipe. 

However, the aspects covered by Snipe are 
not entirely congruent with those covered 
by this manuscript. Some of the kinds 
of neural networks are not supported by 
Snipe, while when it comes to other kinds 
of neural networks, Snipe may have lots 
and lots more capabilities than may ever 
be covered in the manuscript in the form 
of practical hints. Anyway, in my experi- 
ence almost all of the implementation re- 
quirements of my readers are covered well. 
On the Snipe download page, look for the 
section "Getting started with Snipe" - you 
will find an easy step-by-step guide con- 
cerning Snipe and its documentation, as 
well as some examples. 

SIMIPE: This manuscript frequently incor- 
porates Snipe. Shaded Snipe-paragraphs 
like this one are scattered among large 
parts of the manuscript, providing infor- 
mation on how to implement their con- 
text in Snipe. This also implies that 
those who do not want to use Snipe, 
just have to skip the shaded Snipe- 
paragraphs! The Snipe-paragraphs as- 
sume the reader has had a close look at 
the "Getting started with Snipe" section. 
Often, class names are used. As Snipe con- 
sists of only a few different packages, I omit- 
ted the package names within the qualified 
class names for the sake of readability. 



VI 



D. Kriesel - A Brief Introduction to Neural Networks (ZETA2-EN) 



dkriesel.com 



It’s easy to print this 
manuscript 

This text is completely illustrated in 
color, but it can also be printed as is in 
monochrome: The colors of figures, tables 
and text are well-chosen so that in addi- 
tion to an appealing design the colors are 
still easy to distinguish when printed in 
monochrome. 



There are many tools directly 
integrated into the text 

Different aids are directly integrated in the 
document to make reading more flexible: 
However, anyone (like me) who prefers 
reading words on paper rather than on 
screen can also enjoy some features. 



In the table of contents, different 
types of chapters are marked 

Different types of chapters are directly 
marked within the table of contents. Chap- 
ters, that are marked as "fundamental" 
are definitely ones to read because almost 
all subsequent chapters heavily depend on 
them. Other chapters additionally depend 
on information given in other (preceding) 
chapters, which then is marked in the ta- 
ble of contents, too. 



Speaking headlines throughout the 
text, short ones in the table of 
contents 

The whole manuscript is now pervaded by 
such headlines. Speaking headlines are 
not just title-like ("Reinforcement Learn- 
ing"), but centralize the information given 
in the associated section to a single sen- 
tence. In the named instance, an appro- 
priate headline would be "Reinforcement 
learning methods provide feedback to the 
network, whether it behaves good or bad". 
However, such long headlines would bloat 
the table of contents in an unacceptable 
way. So I used short titles like the first one 
in the table of contents, and speaking ones, 
like the latter, throughout the text. 

Marginal notes are a navigational 
aid 

The entire document contains marginal 
notes in colloquial language (see the exam- 
ple in the margin), allowing you to "scan" 
the document quickly to find a certain pas- 
sage in the text (including the titles). 

New mathematical symbols are marked by 
specific marginal notes for easy finding 
(see the example for x in the margin). 

There are several kinds of indexing 

This document contains different types of 
indexing: If you have found a word in 
the index and opened the corresponding 
page, you can easily find it by searching 
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for highlighted text - all indexed words 
are highlighted like this. 

Mathematical symbols appearing in sev- 
eral chapters of this document (e.g. 17 for 
an output neuron; I tried to maintain a 
consistent nomenclature for regularly re- 
curring elements) are separately indexed 
under "Mathematical Symbols", so they 
can easily be assigned to the correspond- 
ing term. 

Names of persons written in small caps 
are indexed in the category "Persons" and 
ordered by the last names. 



Terms of use and license 

Beginning with the epsilon edition, the 
text is licensed under the Creative Com- 
mons Attribution- No Derivative Works 
3.0 Unported License 2 , except for some 
little portions of the work licensed under 
more liberal licenses as mentioned (mainly 
some figures from Wikimedia Commons). 
A quick license summary: 

1. You are free to redistribute this docu- 
ment (even though it is a much better 
idea to just distribute the URL of my 
homepage, for it always contains the 
most recent version of the text). 

2. You may not modify, transform, or 
build upon the document except for 
personal use. 

2 http : / / creativecommons . org/licenses/ 
by- nd/3. 0/ 



3. You must maintain the author’s attri- 
bution of the document at all times. 

4. You may not use the attribution to 
imply that the author endorses you 
or your document use. 

For I’m no lawyer, the above bullet-point 
summary is just informational: if there is 
any conflict in interpretation between the 
summary and the actual license, the actual 
license always takes precedence. Note that 
this license does not extend to the source 
files used to produce the document. Those 
are still mine. 

How to cite this manuscript 

There’s no official publisher, so you need 
to be careful with your citation. Please 
find more information in English and 
German language on my homepage, re- 
spectively the subpage concerning the 
manuscript 3 . 
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